Pockets of predictability due to time-varying persistence in hourly electricity prices

Here we compare AR(3) versus TV-persistence forecasts for 24 hours at h={1,2,6,12,24} hour-ahead forecasts separately

Possible improvements:

  • hyperopt - kernels, bandw, AR choices...
  • explore role of scales (different models using only some scales)
  • explore decomposed inputs and model with decomposition of not only prices but also inputs
In [96]:
using RCall
R"library(tvReg)"

using CSV, DataFrames,GLM
using Distributions, LinearAlgebra, Statistics
using Plots, StatsBase, StatsPlots, Colors
using BSON
using LinearAlgebra
using PrettyTables
using Dates

# load main functions
include("tvPersistence_functions_electricity.jl");
myrainbow=reverse(cgrad(:RdYlBu_7, 7, categorical = true));

I take the hourly data for the first hour

Raw Data

Plot of raw data

In [2]:
data_read=CSV.File("DE1523.csv",missingstring=["NA"],header=true) |> DataFrame;
datesall=Date.(first.(data_read.Column1,10),dateformat"y-m-d");
datesallyr=Date.(first.(data_read.Column1,4),dateformat"y");
In [3]:
v=first.(data_read.Column1,4)
pos=findall(x->x==1,v[2:end].!==v[1:end-1]);

v=first.(data_read.Column1[1:24:end],4)
poshour=findall(x->x==1,v[2:end].!==v[1:end-1]);
In [79]:
plot(data_read.Price,size=(1000,200),title="Hourly prices all",label=false,frame=:box,color=myrainbow[1],titlefontsize=10,xticks=([1;pos],2015:2023))
plot!(fontfamily="serif-roman",titlefontsize=10, xtickfontsize=10,ytickfontsize=10,ylabelfontsize=10)
Out[79]:

Transform data

In [5]:
# transform data
datatransf=[]
a=[]
b=[]
for i=1:24
    datatransf0,a0,b0 = transformation_asinh(data_read.Price[i:24:end]);
    push!(datatransf,datatransf0)
    push!(a,a0)
    push!(b,b0)
end

PLots of transformed data we work with, shaded area is in-sample (rolling window OOS is the rest)

In [78]:
for i in 1:24
    plot(datatransf[i],size=(1000,200),title="Prices - hour $(i-1)",label=false,frame=:box,color=myrainbow[1],
        xticks=([1;poshour],2015:2023),titlefontsize=10)
    plot!(fontfamily="serif-roman",titlefontsize=10, xtickfontsize=10,ytickfontsize=10,ylabelfontsize=10)
    display(plot!([1:1024],fill=myrainbow[1], fillalpha=0.2,linecolor=:transparent,seriestype=:vspan,label=false))
    
end

Forecasts

Here we have ratios TV-EWD to AR(3) losses in terms of RMSE and MAE, so that 0.9 is 10% gain of TV-EWD across all 24 hours at all horizons

In [7]:
folder="results_transformed" # transformed data not tuned (not transformed back)
#folder="results"        # transformed data not tunted kernel
#folder="Old_results"   # nontransformed data

forecasts1=[]
for i in 1:24
    forecast_0 = BSON.load("./$folder/results_electricity_$i-1.bson")
    push!(forecasts1,forecast_0["forecasts"])
end

forecasts2=[]
for i in 1:24
    forecast_0 = BSON.load("./$folder/results_electricity_$i-2.bson")
    push!(forecasts2,forecast_0["forecasts"])
end

forecasts6=[]
for i in 1:24
    forecast_0 = BSON.load("./$folder/results_electricity_$i-6.bson")
    push!(forecasts6,forecast_0["forecasts"])
end

forecasts12=[]
for i in 1:24
    forecast_0 = BSON.load("./$folder/results_electricity_$i-12.bson")
    push!(forecasts12,forecast_0["forecasts"])
end

forecasts24=[]
for i in 1:24
    forecast_0 = BSON.load("./$folder/results_electricity_$i-24.bson")
    push!(forecasts24,forecast_0["forecasts"])
end
In [8]:
losses1=[[rmse(forecasts1[i][:,1],forecasts1[i][:,2]) for i=1:24]'
    [rmse(forecasts1[i][:,1],forecasts1[i][:,4]) for i=1:24]'
    [mae(forecasts1[i][:,1],forecasts1[i][:,2]) for i=1:24]'
    [mae(forecasts1[i][:,1],forecasts1[i][:,4]) for i=1:24]'];

losses2=[[rmse(forecasts2[i][:,1],forecasts2[i][:,2]) for i=1:24]'
    [rmse(forecasts2[i][:,1],forecasts2[i][:,4]) for i=1:24]'
    [mae(forecasts2[i][:,1],forecasts2[i][:,2]) for i=1:24]'
    [mae(forecasts2[i][:,1],forecasts2[i][:,4]) for i=1:24]'];

losses6=[[rmse(forecasts6[i][:,1],forecasts6[i][:,2]) for i=1:24]'
    [rmse(forecasts6[i][:,1],forecasts6[i][:,4]) for i=1:24]'
    [mae(forecasts6[i][:,1],forecasts6[i][:,2]) for i=1:24]'
    [mae(forecasts6[i][:,1],forecasts6[i][:,4]) for i=1:24]'];

losses12=[[rmse(forecasts12[i][:,1],forecasts12[i][:,2]) for i=1:24]'
    [rmse(forecasts12[i][:,1],forecasts12[i][:,4]) for i=1:24]'
    [mae(forecasts12[i][:,1],forecasts12[i][:,2]) for i=1:24]'
    [mae(forecasts12[i][:,1],forecasts12[i][:,4]) for i=1:24]'];

losses24=[[rmse(forecasts24[i][:,1],forecasts24[i][:,2]) for i=1:24]'
    [rmse(forecasts24[i][:,1],forecasts24[i][:,4]) for i=1:24]'
    [mae(forecasts24[i][:,1],forecasts24[i][:,2]) for i=1:24]'
    [mae(forecasts24[i][:,1],forecasts24[i][:,4]) for i=1:24]'];
In [88]:
a=plot([losses1[2,:]./losses1[1,:] losses2[2,:]./losses2[1,:] losses6[2,:]./losses6[1,:] losses12[2,:]./losses12[1,:] losses24[2,:]./losses24[1,:]],
    ylim=[0.5,1.2],ylab="TV-EWD / AR",xlab="hours",label=["h=1" "h=2" "h=6" "h=12" "h=24"],
    frame=:box,title="RMSE Ratio for all hours and horizons",xticks=(1:24,0:23),
    color=[myrainbow[1] myrainbow[3] myrainbow[5] myrainbow[6] myrainbow[7]],linewidth=1)
a=hline!([1],label=false,color=:black)

b=plot([losses1[4,:]./losses1[3,:] losses2[4,:]./losses2[3,:] losses6[4,:]./losses6[3,:] losses12[4,:]./losses12[3,:] losses24[4,:]./losses24[3,:]],
    ylim=[0.5,1.2],ylab="TV-EWD / AR",xlab="hours",label=["h=1" "h=2" "h=6" "h=12" "h=24"],
    frame=:box,title="MAE Ratio for all hours and horizons",xticks=(1:24,0:23),color=[myrainbow[1] myrainbow[3] myrainbow[5] myrainbow[6] myrainbow[7]],
    linewidth=1)
b=hline!([1],label=false,color=:black)

plot(a,b,layout=(1,2),size=(1100,400))
plot!(fontfamily="serif-roman",titlefontsize=10, xtickfontsize=10,ytickfontsize=10,ylabelfontsize=10)
Out[88]:
In [89]:
println("***************************************************************")
println("               RMSE - h=1            MAE - h=1")
println("***************************************************************")
display([["Hours" "AR" "TV-EWD" "AR" "TV-EWD"];
    [[i for i=0:23]'
    losses1]'])
println("***************************************************************")
println("               RMSE - h=2            MAE - h=2")
println("***************************************************************")
display([["Hours" "AR" "TV-EWD" "AR" "TV-EWD"];
    [[i for i=0:23]'
    losses2]'])
println("***************************************************************")
println("               RMSE - h=6            MAE - h=6")
println("***************************************************************")
display([["Hours" "AR" "TV-EWD" "AR" "TV-EWD"];
    [[i for i=0:23]'
    losses6]'])
println("***************************************************************")
println("               RMSE - h=12            MAE - h=12")
println("***************************************************************")
display([["Hours" "AR" "TV-EWD" "AR" "TV-EWD"];
    [[i for i=0:23]'
    losses12]'])
println("***************************************************************")
println("               RMSE - h=24            MAE - h=24")
println("***************************************************************")
display([["Hours" "AR" "TV-EWD" "AR" "TV-EWD"];
    [[i for i=0:23]'
    losses24]'])

***************************************************************
               RMSE - h=1            MAE - h=1
***************************************************************

25×5 Matrix{Any}:
   "Hours"   "AR"      "TV-EWD"   "AR"      "TV-EWD"
  0.0       0.526434  0.533566   0.347887  0.349313
  1.0       0.553079  0.561537   0.363893  0.365389
  2.0       0.559725  0.56956    0.370766  0.37164
  3.0       0.572172  0.579087   0.379384  0.380331
  4.0       0.568053  0.571828   0.372839  0.374583
  5.0       0.554842  0.549087   0.361789  0.355696
  6.0       0.542712  0.493261   0.37472   0.325739
  7.0       0.536944  0.453491   0.382648  0.300738
  8.0       0.533225  0.45383    0.381052  0.303171
  9.0       0.542747  0.482179   0.3812    0.327201
 10.0       0.567864  0.520038   0.397229  0.357886
 11.0       0.595019  0.553489   0.410382  0.380735
 12.0       0.629717  0.59203    0.431915  0.406141
 13.0       0.682324  0.631605   0.470175  0.432396
 14.0       0.687184  0.629578   0.470525  0.430211
 15.0       0.632229  0.58863    0.42983   0.397937
 16.0       0.547959  0.515829   0.36972   0.349974
 17.0       0.437367  0.416238   0.29719   0.286195
 18.0       0.39222   0.38162    0.272488  0.263733
 19.0       0.397283  0.38726    0.283976  0.274527
 20.0       0.406738  0.404822   0.285647  0.280782
 21.0       0.418913  0.423915   0.284475  0.286823
 22.0       0.401687  0.41329    0.266843  0.276665
 23.0       0.462973  0.471338   0.303313  0.311765
***************************************************************
               RMSE - h=2            MAE - h=2
***************************************************************

25×5 Matrix{Any}:
   "Hours"   "AR"      "TV-EWD"   "AR"      "TV-EWD"
  0.0       0.483505  0.473389   0.337281  0.329029
  1.0       0.508312  0.497958   0.354476  0.346431
  2.0       0.516192  0.505351   0.360139  0.351148
  3.0       0.530444  0.516551   0.369279  0.36002
  4.0       0.531663  0.514169   0.369089  0.355746
  5.0       0.524923  0.497476   0.363538  0.339974
  6.0       0.513734  0.436847   0.367805  0.298212
  7.0       0.511083  0.395694   0.374642  0.273528
  8.0       0.509536  0.399934   0.374378  0.279424
  9.0       0.513412  0.426191   0.37149   0.30149
 10.0       0.540642  0.463776   0.389031  0.330106
 11.0       0.566411  0.493776   0.404847  0.348949
 12.0       0.59772   0.528171   0.426603  0.37392
 13.0       0.652342  0.565199   0.467758  0.399493
 14.0       0.659736  0.563998   0.470739  0.398122
 15.0       0.608338  0.530134   0.431002  0.369556
 16.0       0.530252  0.469746   0.371076  0.327509
 17.0       0.427689  0.385487   0.300106  0.27188
 18.0       0.376918  0.347496   0.271729  0.248582
 19.0       0.376624  0.347265   0.27757   0.251324
 20.0       0.382763  0.362907   0.279952  0.259051
 21.0       0.390544  0.378453   0.277549  0.265919
 22.0       0.374209  0.370226   0.260689  0.256851
 23.0       0.420773  0.414434   0.293804  0.287789
***************************************************************
               RMSE - h=6            MAE - h=6
***************************************************************

25×5 Matrix{Any}:
   "Hours"   "AR"      "TV-EWD"   "AR"      "TV-EWD"
  0.0       0.45316   0.399731   0.326419  0.289902
  1.0       0.473584  0.416746   0.344545  0.303588
  2.0       0.481432  0.423384   0.351941  0.308657
  3.0       0.492664  0.43104    0.361346  0.315615
  4.0       0.493461  0.427323   0.362301  0.311316
  5.0       0.488806  0.414568   0.357297  0.297204
  6.0       0.467251  0.362751   0.341016  0.251879
  7.0       0.456068  0.325906   0.333076  0.228267
  8.0       0.455702  0.326776   0.333729  0.23375
  9.0       0.462778  0.350146   0.339408  0.256487
 10.0       0.488149  0.381064   0.358391  0.281535
 11.0       0.509498  0.405371   0.373325  0.298445
 12.0       0.533     0.428056   0.390322  0.315963
 13.0       0.575298  0.457183   0.423984  0.338347
 14.0       0.580439  0.457844   0.426537  0.340095
 15.0       0.541885  0.434731   0.395904  0.322514
 16.0       0.477733  0.391314   0.345333  0.291039
 17.0       0.401203  0.335208   0.289228  0.24849
 18.0       0.360055  0.304407   0.265097  0.226443
 19.0       0.361325  0.303652   0.268535  0.225928
 20.0       0.37026   0.317604   0.272265  0.232662
 21.0       0.377768  0.331347   0.273115  0.242679
 22.0       0.363869  0.325735   0.25871   0.236717
 23.0       0.401584  0.357096   0.289202  0.26137
***************************************************************
               RMSE - h=12            MAE - h=12
***************************************************************

25×5 Matrix{Any}:
   "Hours"   "AR"      "TV-EWD"   "AR"      "TV-EWD"
  0.0       0.482285  0.372016   0.350691  0.266857
  1.0       0.500253  0.384481   0.366577  0.278821
  2.0       0.507449  0.389718   0.373814  0.282848
  3.0       0.51935   0.399036   0.383752  0.289272
  4.0       0.521779  0.395334   0.385951  0.285272
  5.0       0.523896  0.384215   0.386409  0.274843
  6.0       0.513533  0.343884   0.374566  0.239342
  7.0       0.506443  0.311901   0.367978  0.218314
  8.0       0.506413  0.311462   0.369879  0.223045
  9.0       0.50739   0.330909   0.370107  0.243437
 10.0       0.526032  0.354801   0.385749  0.263781
 11.0       0.542865  0.375539   0.39725   0.277097
 12.0       0.563283  0.393152   0.412538  0.288426
 13.0       0.605188  0.417747   0.443965  0.308935
 14.0       0.611863  0.419894   0.44865   0.314144
 15.0       0.576726  0.404607   0.421575  0.302343
 16.0       0.51284   0.372466   0.375041  0.277693
 17.0       0.440853  0.327844   0.321596  0.243795
 18.0       0.403795  0.302646   0.297828  0.224035
 19.0       0.410985  0.304974   0.30106   0.222467
 20.0       0.41521   0.311637   0.30209   0.223641
 21.0       0.416643  0.321635   0.303043  0.233381
 22.0       0.397073  0.31468    0.285092  0.225881
 23.0       0.433822  0.339532   0.315058  0.243178
***************************************************************
               RMSE - h=24            MAE - h=24
***************************************************************

25×5 Matrix{Any}:
   "Hours"   "AR"      "TV-EWD"   "AR"      "TV-EWD"
  0.0       0.548622  0.349383   0.398856  0.250913
  1.0       0.564498  0.35988    0.410916  0.261386
  2.0       0.571099  0.364804   0.417064  0.264613
  3.0       0.58608   0.376725   0.429456  0.2728
  4.0       0.59177   0.373547   0.433847  0.269837
  5.0       0.601529  0.362637   0.442881  0.261411
  6.0       0.596269  0.328285   0.433053  0.234032
  7.0       0.58997   0.301395   0.421318  0.21328
  8.0       0.591883  0.299298   0.426404  0.214918
  9.0       0.591591  0.319308   0.428133  0.23415
 10.0       0.608077  0.343202   0.442038  0.251233
 11.0       0.619723  0.361543   0.452167  0.264599
 12.0       0.637434  0.376813   0.468656  0.276689
 13.0       0.672797  0.397899   0.496531  0.293171
 14.0       0.678991  0.403224   0.501381  0.3003
 15.0       0.651117  0.389874   0.479191  0.292263
 16.0       0.59035   0.363797   0.431316  0.272874
 17.0       0.519736  0.329642   0.376137  0.245811
 18.0       0.481697  0.305782   0.3506    0.226267
 19.0       0.493949  0.310184   0.358143  0.224976
 20.0       0.494453  0.309146   0.361776  0.221204
 21.0       0.489609  0.315194   0.361782  0.229078
 22.0       0.462659  0.306677   0.340172  0.22137
 23.0       0.49924   0.323196   0.367203  0.232965

Pockets of predictability for h=1

pockets of predictability identified for all hours, those in darker blue area are not spuriously (or randomly) generated, we use simple persistent process in the bootstrap

Areas simply point to periods when TV-EWD model improves accuracy over AR(3)

In [62]:
poshouroos=poshour.-1024;
poshouroos=poshouroos[3:end];
In [98]:
bw=0.03;
for i in 1:24
    display(plot_pockets(forecasts1[i][:,2].-forecasts1[i][:,1],forecasts1[i][:,4].-forecasts1[i][:,1],bw,"Pockets of predictability - hour $(i-1)"))
end

Example of forecasts for hour 8 (h=1 and h=6)

Lubos's comparison period

naive is 9,2559 LEAR 4.4655

In [31]:
[mean([mae(forecasts1[i][616:616+554-1,1],forecasts1[i][616:616+554-1,2]) for i=1:24])
mean([mae(forecasts1[i][616:616+554-1,1],forecasts1[i][616:616+554-1,4]) for i=1:24])]
Out[31]:
2-element Vector{Float64}:
 8.802558373515488
 7.9989122002823585
In [283]:
v=first.(data_read.Column1[9:24:end][1024:end],4)
poshouroos=findall(x->x==1,v[2:end].!==v[1:end-1]);

plot(forecasts1[9][:,[1;2;4]],size=(1200,220),label=["data" "AR forecast" "TV-EWD forecast"],frame=:box,title="h=1 forecasts",
    color=[myrainbow[1] myrainbow[5] myrainbow[7]],titlefontsize=10,xticks=(poshouroos,2018:2023))
Out[283]:
In [284]:
plot(forecasts6[9][:,[1;2;4]],size=(1200,220),label=["data" "AR forecast" "TV-EWD forecast"],frame=:box,title="h=6 forecasts",
    color=[myrainbow[1] myrainbow[5] myrainbow[7]],titlefontsize=10,xticks=(poshouroos,2018:2023))
Out[284]: